Method and apparatus for image processing based on a mapping function

ABSTRACT

A method for processing a digital input image data indexed to represent positions on a display such that the digital data is indicative of an intensity value for each position. The intensity value for each position is transformed to generate an enhanced value, using a mapping function which transforms the same intensity value of input data to essentially the same intensity value of output data in the luminance channel.

FIELD OF THE INVENTION

The present invention relates generally to image processing, and moreparticularly to a method and apparatus for increasing details in darkregions of an image while maintaining brightness and color constancy.

BACKGROUND OF THE INVENTION

Image processing systems are at the heart of digital image revolution.These systems process the captured digital image to enhance the clarityand details of the image using image processing algorithms. Suchalgorithms result in images that are substantially more accurate anddetailed than previously achieved using older analog methods.

There remains, however, a substantial difference between how an image isperceived by a person and an image captured and reproduced on a displaymedium. Despite the improvements gained by conventional digital imageprocessing systems, such systems are still deficient in reproducing animage with the same level of detail, color constancy, and lightness ofan actual scene as the eye, brain, and nervous system of a human being.This is due in part because the human nervous system has a greaterdynamic range compression than is available on current digital systems.Dynamic range compression refers to the ability to distinguish varyinglevels of light.

The human eye has a dynamic range compression of approximately 1000:1,which means that the human eye can distinguish approximately 1000 levelsof light variations. By way of contrast, digital image systems typicallyuse only eight bits per pixel which allows for a dynamic rangecompression of only 255:1. As a result, a digital image reproduced as aphotograph would have far less detail in the darker and brighter regionsof the photograph as compared to the actual scene perceived by a viewer.

Many techniques have been developed to compensate for this lightingdeficiency. These techniques can be separated into two broad categories:(1) power law or nonlinear techniques (“non-linear transforms”); and (2)retinex techniques. Each have their respective limitations however.

Non-linear techniques use a non-linear relationship to expand oneportion of the dynamic range while compressing another. These techniquesgenerally enhance details in the darker regions at the expense of detailin the brighter regions. One problem associated with these non-linearsystems is that they provide greater distinctions between pixelsregardless of what lightness value a pixel may have been originallyassigned. This results in the brighter areas which already have a“washedout” appearance to become even more washed-out. Although thesetechniques result in better details in the darker regions, they do so atthe expense of the brighter areas of the digital image. Further, thesemethods cannot handle contours and abrupt boundaries well.

Retinex techniques variations increase or decrease the luminance valuefor a pixel based on the luminance values of surrounding pixels. Thesetechniques are particularly useful for enhancing boundaries betweenlighter and darker regions of an image. However, such techniques areunsatisfactory for a number of reasons. In one technique, a largeuniform zones in the image are grayed out (i.e., shading effect). Inanother technique a shift in color occurs in some images (i.e., colordistortion) and is computationally intensive.

SUMMARY OF THE INVENTION

The present addresses the above shortcomings. It is an object of thepresent invention to provide a method of improving a digital image sothat the image appears similar to what is perceived by human vision inall kinds and levels of lighting across the entire scene. As such, inone embodiment the present invention provides a method and apparatus forprocessing an image wherein a path is defined as a set of locationsdistributed over an entire input data. A tone mapping function iscreated by comparing intensity values based on retinex path computationscheme without selecting paths, which transforms each entry of inputdata into the enhanced value.

In another embodiment, the present invention provides an example imageprocessing method based on retinex theory, with at least the followingdifferences compared with the conventional retinex-based algorithms: 1)the example method operates only on a luminance channel while theconventional methods operate on three different color channels, 2) theexample method does not cause shading effect; 3) the example method doesnot cause color distortion; and 4) the example method utilizes a tonemapping function which transforms the same intensity value of input datato the same intensity value of output data, whereas conventional retinexmethods map the same intensity values to the different values accordingto local and global information of the input data.

Other embodiments, features and advantages of the present invention willbe apparent from the following specification taken in conjunction withthe following drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a functional block diagram of an embodiment of an imageprocessing according to the present invention.

FIG. 2 shows a functional block diagram of an embodiment of an imageenhancer according to the present invention.

FIG. 3 shows example computation steps performed by an image enhanceraccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

A example method and apparatus of improving details in a digital imageaccording to an embodiment of the present invention is described.Accordingly, details for a digital image are enhanced while brightnessand color constancy are preserved. At a given point (e.g., pixel) in theimage, a determination is made as to whether the pixel “path elements”constituting a path are brighter or darker than the point. The output ofthe given point is adjusted to generate an enhanced value from averagingthe outputs of path-computations selected against the given point.

FIG. 1 shows a block diagram of an image processing system 10 accordingto an embodiment of the present invention. An image is captured anddigitized in block 100 according to well-known techniques, wherein thedigitized image is represented by discrete areas referred to as pixels.In one example, the digitized image comprises three color channels,which are Red, Green, and Blue channels (also known as RGB). The colorchannels may be transformed into other color spaces such as CIE and YUV(Y represents luminance, U a first color difference, and V a secondcolor difference).

According to the embodiment of the present invention described herein,the digitized image is transformed to YUV space in a transformationblock 102. Each pixel is assigned a Y UV value. The Y value controls thebrightness for that particular pixel. Conventional systems typicallyutilize eight bits to represent the Y value due to bandwidth efficiencyand memory design considerations. Therefore, conventional imageprocessing systems assign each pixel a Y value somewhere in the range of0 to 255, with 0 representing the darkest luminance and 255 representingthe brightest luminance.

Then, in an image enhancement block 104, image enhancement techniquesare used to emphasize and sharpen image features for display. Suchenhancement techniques operate in the spatial domain by manipulating thepixel data, or operate in the frequency domain by modifying the spectralcomponents. The example enhancement technique according to the presentinvention, operates in the spatial domain, and more particularly,applies a transform only on the luminance value Y.

The example technique essentially enhances the details in the darkerregions of the digitally recorded images without washing out the detailsat the brighter ends, thereby making the digitally recorded images morerealistic with respect to an actual viewer. The example techniquefurther reduces graying out of large uniform zones of color as occursusing conventional techniques. The example technique also reduces colorshift as it operates only on the Y channel. In addition, the exampletechnique is computationally efficient and fast as it operates only onthe Y channel.

An inverse color transformation block 106 then transforms the enhancedimage from YUV color channels back to RGB color channels using arotation matrix. The enhanced RGB image is then output by block 108.

FIG. 2 is a block diagram of an image enhancer 200 in accordance with anembodiment of the present invention, comprising a tone mapping functiongenerator 202, wherein two color difference components (C1 and C2) passthrough the enhancer 200. FIG. 3 shows the details of an embodiment ofthe mapping function generator 202, illustrating a path in the enhancer200 to be calculated. According to the present invention, select pathsand their computations is not required. However, to simplifyunderstanding of the present invention, an example is presented. It ispreferred that the number of elements in a path (“path length”), be asmany as possible. Further, a number of paths may be selected andaveraged after computing same measures for each path to obtain a updatedvalue for a pixel value.

One example path computation is presented because same computation isapplied to the rest of paths if another path exists. Most imageenhancement algorithms usually perform on Log-scaled space forcomputation efficiency and more dynamic range compression issue.However, for explanatory convenience, all descriptions here initiallyutilize conventional decimal space.

Referring to the example in FIG. 3, starting at a random pixel denotedas Y⁰(x, y) and following a random path (a sequence of pixels) to an endposition Y^(n)(x, y), along the way a sequence of pixels denoted asY¹(x, y), Y²(x, y), . . . , are visited. An intermediate output F_(p)^(n) (x, y) is defined at the n^(th) element in the path. It is worthnoting that each alphabet symbol, p, corresponding to intensity valueY^(n)(x, y) at n^(th) element is not intensity value but elementposition of a path in xy image plane. The example path computation is asfollows.

1. Assign an initial value α₀ to F_(p) ⁰ (x, y), wherein the initialvalue α₀ is maximum intensity value of an input image data, where:F _(p) ⁰(x, y)=α₀  (1)2. Computer the ratio of Y¹(x, y) to Y⁰(x, y) at position k according torelation (2) below:Y¹(x, y)/Y⁰(x, y)  (2)

wherein this value is assigned to F_(k) ¹ (x, y).3. Move to a position j, and determine the ratio of Y²(x, y) to Y¹(x, y)by taking the product of the value F_(k) ¹ (x, y) according to relation(3) below: $\begin{matrix}{{\frac{Y^{2}\left( {x,y} \right)}{Y^{1}\left( {x,y} \right)} \cdot \frac{Y^{1}\left( {x,y} \right)}{Y^{0}\left( {x,y} \right)}} = {\frac{Y^{2}\left( {x,y} \right)}{Y^{0}\left( {x,y} \right)}.}} & (3)\end{matrix}$

This value is assigned to F_(j) ² (x, y). Then the steps are repeatedfor the next location/point in the path.

4. At the end of the path, the ratio has been computed between the startposition and each location along the path and assigned to (i.e., storedat) the corresponding position. This process is repeated for manydifferent paths with different start and end points. Then, a history ofa location's ratio to many different pixels is recorded, and all theratios are averaged to obtain an estimate of the lightness (e.g.,intensity) at a location (known as an averaging procedure).

In summary, the key elements in the retinex computation are to determinea ratio followed by a product along a random path and to average theresults over many different paths to obtain estimate of the lightness ata location.

In addition, some versions of the algorithm supplement the ratio andproduct operations with two further operations: a thresholding operationand a resetting operation. According to threshold-reset operations, anintermediate value F_(n) ^(i) (x, y) is modified according to relation(4) below: $\begin{matrix}{{F_{i}^{n}\left( {x,y} \right)} = \left\{ \begin{matrix}{{\frac{I^{n}\left( {x,y} \right)}{I^{n - 1}}{if}\quad{\frac{I^{n}\left( {x,y} \right)}{I^{n - 1}\left( {x,y} \right)} \cdot F_{l}^{n - 1}}} < F_{i}^{0}} \\{F_{i}^{0}{o.w.}}\end{matrix} \right.} & (4)\end{matrix}$

where O.W. indicates “otherwise”, F_(l) ^(n−1) (x, y) is theintermediate value at the previous n−1 position. The intermediate valuesF_(n) ^(i) (x, y) have a history that accumulates a comparison resultwith path elements passing through the position i. A pixel location isaffected by the pixel locations that a path goes through in an entireinput image. It is found that a pixel value is updated by averaging theintermediate values in paths, with respect to global information of aninput image, in which the corresponding reference input values arebrighter than the current pixel reference value.

An example image processing method according to the present inventionapproximates the retinex algorithm. Such an image processing method canreduce computational complexity and speed up the process because it neednot consider the path elements. As noted, the output of a pathcomparison can be the average of the path elements which are brighterthan a pixel value at the output position. If the pixel position hasmultiple outputs for multiple paths, the final output is the average ofthose multiple outputs from multiple paths. It is important to note thatthe output at a position depends on the length of a path and the orderof elements in the path. It is preferred that the number of elements ina path be as many as possible and as random as possible over entireinput data.

The conditional probability of an event B in relation to an event A isthe probability that event B occurs given that the event A has alreadyoccurred, as:P_(r)(B|A)  (5)

wherein the conditional probability of an event is determined bydividing both sides of a multiplication rule by P_(r)(A) according torelation (6) below: $\begin{matrix}{{P_{r}\left( B \middle| A \right)} = \frac{P_{r}\left( {B\bigcap A} \right)}{P_{r}(A)}} & (6)\end{matrix}$

Relation (6) can also be expressed by relation (7) below:$\begin{matrix}{{{P_{r}\left( B \middle| A \right)} = {\sum\limits_{i = 1}^{N}{P_{r}\left( B \middle| A_{i} \right)}}},} & (7)\end{matrix}$

where A_(i) is a disjointed subset of A such that A₁∩A₂∩ . . . ∩A_(N)=0;and $\begin{matrix}{A = {\sum\limits_{i = 1}^{N}{A_{i}.}}} & (8)\end{matrix}$

Based on foregoing statements, a variant framework of retinex methodaccording to an embodiment of the present invention is to obtain amapping function {tilde over (G)}(x_(i)), where x_(i) is the bin indexof the mapping function. Computing the mapping function is as follows.

1. Assign initial values β·α₀+(1−β)·x_(i) to each bin x_(i) of G(x_(i))wherein the value α₀ is a maximum value of an input image, and the valueβ is a control gain. The range of the control gain varies from 0 to 1(e.g., β=0.5).

2. From relations (4) and (7) above, if a current position i and itsvalue x_(i) are affected by the previous value x_(j) over an entireimage, an intermediate output of the mapping function G(x) can beexpressed according to relation (9) below: $\begin{matrix}{{G\quad\left( x_{i} \right)} = \left\{ \begin{matrix}{{{{\frac{x_{i}}{x_{j}} \cdot G}\quad\left( x_{j} \right)\quad{if}\quad x_{i}} < x_{j}},} \\{G\quad\left( x_{j} \right)\quad{o.w.}}\end{matrix} \right.} & (9)\end{matrix}$

where the value of the mapping function G(x_(i)) is reset to 1 when theprevious pixel value is darker than the current pixel value. Therefore,the expected value at the current position {tilde over (G)}(x_(i)) isaccording to relation (10) below: $\begin{matrix}{{\overset{\sim}{G}\quad\left( x_{i} \right)} = {{\sum\limits_{x_{j} = 1}^{x_{i}}{G\quad{\left( x_{j} \right) \cdot {P_{r}\left( x_{j} \right)}}}} + {\sum\limits_{x_{j} = x_{i}}^{MAX\_ BIN}{{\frac{x_{i}}{x_{j}} \cdot G}\quad{\left( x_{j} \right) \cdot {{P_{r}\left( x_{j} \right)}.}}}}}} & (10)\end{matrix}$In relation (10), the value P_(r)(x_(j)) is the probability of a valuex_(j) that can be obtained from the normalized histogram of an inputdata, and MAX_BIN is the maximum bin number allowed to create themapping function. Note that the intermediate output of the mappingfunction, G(x_(i)) is replaced by {tilde over (G)}(x_(i)) when theprocess moves from the current bin index to the next bin index.3. If iterative processing is necessary, the above Step 2 is repeated,wherein the initial values of the next iteration are the output of thecurrent step. In one implementation, two or three iterations providesatisfactory results. The final output image data Y′(x, y) from inputimage data Y (x, y) is according to relation (11) below:Y′(x, y)={tilde over (G)}(Y(x, y)).  (11)

As such, an input image is transformed to an output image by using atone mapping function (e.g., in block 202 of FIG. 2) according to thepresent invention, wherein the mapping function is dependent on thecontents of the input image.

In summary, the input image is initially represented by digital dataindexed to represent positions on a display. The indexed digital data isindicative of an intensity value Y (x, y) for each position (x, y). Theintensity value for each position is transformed to generate an enhancedvalue as: $\begin{matrix}{{\overset{\sim}{G}\quad\left( x_{i} \right)} = {{\sum\limits_{x_{j} = 1}^{x_{i}}{\overset{\sim}{G}\quad{\left( x_{j} \right) \cdot {P_{r}\left( x_{j} \right)}}}} + {\sum\limits_{x_{j} = {x_{i} + 1}}^{MAX\_ BIN}{{\frac{x_{i}}{x_{j}} \cdot \overset{\sim}{G}}\quad{\left( x_{j} \right) \cdot {{P_{r}\left( x_{j} \right)}.}}}}}} & \quad\end{matrix}$where {tilde over (G)}(l) is a mapping function for an intensity valuel, P_(r)(l) is the probability of a value l that can be obtained fromthe normalized histogram of input data, and MAX BIN is the number ofintensity values to be considered (e.g., 256). As such, in the examplemainly the luminance (Y-component) values are enhanced.

While this invention is susceptible of embodiments in many differentforms, there are shown in the drawings and will herein be described indetail, preferred embodiments of the invention with the understandingthat the present disclosure is to be considered as an exemplification ofthe principles of the invention and is not intended to limit the broadaspects of the invention to the embodiments illustrated. Theaforementioned example architectures above according to the presentinvention, can be implemented in many ways, such as program instructionsfor execution by a processor, as logic circuits, as ASIC, as firmware,etc., as is known to those skilled in the art. Therefore, the presentinvention is not limited to the example embodiments described herein.

The present invention has been described in considerable detail withreference to certain preferred versions thereof; however, other versionsare possible. Therefore, the spirit and scope of the appended claimsshould not be limited to the description of the preferred versionscontained herein.

1. A method for processing an image, comprising the steps of: providing digital input image data indexed to represent positions on a display, said digital data being indicative of an intensity value for each position; and transforming said intensity value for each position to generate an enhanced value, using a mapping function which transforms the same intensity value of input data to essentially the same intensity value of output data.
 2. The method of claim 1, wherein said intensity value comprises a luminance value.
 3. The method of claim 2, wherein the mapping function operates on the luminance channel only.
 4. The method of claim 3, wherein the mapping function operates on the luminance channel only essentially without causing shading or color distortion effects.
 5. The method of claim 1, wherein the step of transformation further includes the steps of defining a path as a set of positions distributed over the input image data, wherein the mapping function is based on comparing intensity values using a retinex path computation scheme without selecting paths, such that the intensity value for each position is enhanced.
 6. The method of claim 1, wherein: said digital data is indicative of an intensity value Y(x, y) for each position (x, y) in a luminance channel; the step of transforming further includes the steps of transforming said intensity value for each position to generate an enhanced value Y′(x, y) using the mapping function: ${\overset{\sim}{G}\quad\left( x_{i} \right)} = {{\sum\limits_{x_{j} = 1}^{x_{i}}{G\quad{\left( x_{j} \right) \cdot {P_{r}\left( x_{j} \right)}}}} + {\sum\limits_{x_{j} = {x_{i} + 1}}^{MAX\_ BIN}{{\frac{x_{i}}{x_{j}} \cdot G}\quad{\left( x_{j} \right) \cdot {P_{r}\left( x_{j} \right)}}}}}$ where {tilde over (G)}(l) is a final mapping function for an intensity value l, G(l) is an intermediate mapping function, P_(r)(l) is the probability of an intensity value l that can be obtained from the normalized histogram of said digital data, and MAX_BIN is the number of said intensity values to be considered.
 7. The method of claim 6, wherein said intensity value comprises a luminance value.
 8. The method of claim 6, wherein said transformed intensity value Y′(x, y) is a mapped value for said each input digital value Y(x, y) in accordance with Y′(x, y)={tilde over (G)}(Y(x, y)) wherein the mapping function is dependent on the contents of an input data.
 9. The method of claim 6, wherein said maximum bin number MAX_BIN is same as the number of bits allowed to express said intensity value.
 10. The method of claim 6, wherein said probability P_(r)(l) of said intensity value l is calculated based on said normalized histogram of said digital data.
 11. The method of claim 10, wherein the value of said normalized histogram at each bin l is computed using the value of histogram at said each bin l divided by the size of said digital data.
 12. The method of claim 6, further comprising the steps of assigning initial values for the mapping function as: G(x _(i))=β·α₀+(1−β)·x _(i) where the value α₀ is a maximum intensity value of said input digital data, and the value β is an initial control gain.
 13. The method of claim 12 wherein said control gain β can vary from 0 to
 1. 14. The method of claim 6, wherein the intermediate value of said mapping function is defined as: ${G\quad\left( x_{i} \right)} = \left\{ \begin{matrix} {{{{\frac{x_{i}}{x_{j}} \cdot G}\quad\left( x_{j} \right)\quad{if}\quad x_{i}} < x_{j}},} \\ {G\quad\left( x_{j} \right)\quad{o.w.}} \end{matrix} \right.$ wherein the mapping function G(x_(i)) is reset to 1 when a previous value x_(j) at a position j is darker than the current value x_(i) at a position i, and said intermediate output of said mapping function is replaced by {tilde over (G)}(x_(i)) when a process moves from the current bin index x_(i) to the next bin index x_(i)+1.
 15. The method of claim 6, wherein said mapping function {tilde over (G)}(l) for each bin index l is performed for 1≦l≦MAX_BIN.
 16. A method of processing a digital image having pixels, comprising the steps of: transforming RGB channels in the image to a luminance Y channel and two color difference channels; defining a path as a set of pixel luminance positions distributed over the input image; approximating path computations without visiting an ordered sequence of said luminance positions to reach at a target position only in said luminance channel; providing a mapping function using probability of the luminance value at the target position based on a normalized histogram of said input image; transforming said target position luminance value using said mapping function, such that the enhanced value has essentially the same intensity value as the input value and inverting the enhanced luminance value and said two color difference channels into RGB channels as an enhance output image.
 17. A system for processing a digital input image data indexed to represent positions on a display, said digital data being indicative of an intensity value for each position, comprising: an image enhancer that transforms said intensity value for each position to generate an enhanced value, the image enhancer including a tone mapping function which transforms the same intensity value of input data to essentially the same intensity value of output data.
 18. The system of claim 17, wherein said intensity value comprises a luminance value.
 19. The system of claim 18, wherein the mapping function operates on the luminance channel only.
 20. The system of claim 19, wherein the mapping function operates on the luminance channel only, essentially without causing shading or color distortion effects.
 21. The system of claim 17, wherein the image enhancer further defines a path as a set of positions distributed over the input image data, wherein the mapping function is based on comparing intensity values using a retinex path computation scheme without selecting paths, such that the intensity value for each position is enhanced.
 22. The system of claim 17, wherein: said digital data is indicative of an intensity value Y(x, y) for each position (x, y) in a luminance channel; the enhancer further transforms said intensity value for each position to generate an enhanced value Y′(x, y) using the mapping function: ${\overset{\sim}{G}\left( x_{i} \right)} = {{\sum\limits_{x_{j} = 1}^{x_{i}}{{G\left( x_{j} \right)} \cdot {P_{r}\left( x_{j} \right)}}} + {\sum\limits_{x_{j} = {x_{i} + 1}}^{MAX\_ BIN}{\frac{x_{i}}{x_{j}} \cdot {G\left( x_{j} \right)} \cdot {P_{r}\left( x_{j} \right)}}}}$ where {tilde over (G)}(l) is a final mapping function for an intensity value l, G(l) is an intermediate mapping function, P_(r)(l) is the probability of an intensity value l that can be obtained from the normalized histogram of said digital data, and MAX_BIN is the number of said intensity values to be considered.
 23. The system of claim 22, wherein said intensity value comprises a luminance value.
 24. The system of claim 22, wherein said transformed intensity value Y′(x, y) is a mapped value for said each input digital value Y(x, y) in accordance with Y′(x, y)={tilde over (G)}(Y(x, y)) wherein the mapping function is dependent on the contents of an input data.
 25. The system of claim 22, wherein said maximum bin number MAX_BIN is same as the number of bits allowed to express said intensity value.
 26. The system of claim 22, wherein said probability P_(r)(l) of said intensity value l is calculated based on said normalized histogram of said digital data.
 27. The system of claim 26, wherein the value of said normalized histogram at each bin l is computed using the value of histogram at said each bin l divided by the size of said digital data.
 28. The system of claim 22, wherein the enhancer further assigns initial values for the mapping function as: G(x _(i))=β·α₀+(1−β)·x _(i) where the value α₀ is a maximum intensity value of said input digital data, and the value β is an initial control gain.
 29. The system of claim 28, wherein said control gain β can vary from 0 to
 1. 30. The system of claim 22, wherein the intermediate value of said mapping function is defined as: ${G\left( x_{i} \right)} = \left\{ \begin{matrix} {{{{\frac{x_{i}}{x_{j}} \cdot {G\left( x_{j} \right)}}\quad{if}\quad x_{i}} < x_{j}},} \\ {{G\left( x_{j} \right)}{o.w.}} \end{matrix} \right.$ wherein the mapping function G(x_(i)) is reset to 1 when a previous value x_(j) at a positions is darker than the current value x_(i) at a position i, and said intermediate output of said mapping function is replaced by {tilde over (G)}(x_(i)) when a process moves from the current bin index x_(i) to the next bin index x_(i)+1.
 31. The system of claim 22, wherein said mapping function {tilde over (G)}(l) for each bin index l is performed for 1≦l≦μMAX_BIN.
 32. A system for processing a digital image having pixels, comprising: (a) a color transformer that transforms RGB channels in the image to a luminance Y channel and two color difference channels; (b) an image enhancer that: defines a path as a set of pixel luminance positions distributed over the input image; approximates path computations without visiting an ordered sequence of said luminance positions to reach at a target position only in said luminance channel; utilizes a mapping function that uses probability of the luminance value at the target position based on a normalized histogram of said input image; transforms said target position luminance value using said mapping function to an enhanced value, such that the enhanced value has essentially the same intensity value as the input value; and (c) an inverse color transformer that inverts the enhanced luminance value and said two color difference channels into RGB channels as an enhance output image. 